Compatible Stereoscopic Video Delivery

ABSTRACT

Stereoscopic images are subsampled and placed in a “checkerboard” pattern in an image. The image is encoded in a monoscopic video format. The monoscopic video is transmitted to a device where the “checkerboard” is decoded. Portions of the checkerboard (e.g., “black” portions) are used to reconstruct one of the stereoscopic images and the other portion of the checkerboard (e.g., “white” portions) are used to reconstruct the other image. The subsamples are, for example, taken from the image in a location coincident to the checkerboard position in which the subsamples are encoded.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/285,007, filed on Feb. 25, 2019, which is a continuation of U.S.patent application Ser. No. 16/044,836, filed on Jul. 25, 2018, (nowU.S. Pat. No. 10,264,235), which is a continuation of U.S. patentapplication Ser. No. 15/969,349, filed on May 2, 2018, (now U.S. Pat.No. 10,038,891), which is a continuation of U.S. patent application Ser.No. 15/876,975, filed on Jan. 22, 2018, (now U.S. Pat. No. 9,992,476),which is a continuation of U.S. patent application Ser. No. 15/808,675,filed on Nov. 9, 2017, (now U.S. Pat. No. 9,912,931), which is acontinuation of U.S. patent application Ser. No. 15/625,854, filed onJun. 16, 2017, (now U.S. Pat. No. 9,843,785), which is continuation ofU.S. patent application Ser. No. 14/941,992, filed on Nov. 16, 2015,(now U.S. Pat. No. 9,712,801), which is a continuation of U.S. patentapplication Ser. No. 13/054,855, filed on Jan. 19, 2011, (now U.S. Pat.No. 10,136,118), which is the national stage entry for PCT ApplicationSer. No. PCT/US2009/050809, filed on Jul. 16, 2009, which claims thebenefit of priority to U.S. Provisional Patent Application No.61/082,217, filed on Jul. 20, 2008, all of which are hereby incorporatedby reference in their entirety.

TECHNOLOGY

The present invention relates to video coding and more particularly tostereoscopic video coding.

BACKGROUND Description of Related Art

In recent years, content providers have become considerably interestedin the delivery of stereoscopic (3D) content into the home. Thisinterest is driven by the increased popularity and production of 3Dmaterial, but also the emergence of several stereoscopic devices thatare already available to the consumer. Although several systems havebeen proposed on the delivery of stereoscopic material to the home thatcombine specific video view “arrangement” formats with, primarily,existing video compression technologies such as ISO MPEG-2, MPEG-4AVC/ITU-T H.264, and VC-1, these systems do not provide any informationon how the video encoding process should be performed. This hasconsequently resulted in poorly designed stereo video encoding solutionswith subpar performance, which has been detrimental in the adoption ofsuch systems.

SUMMARY OF THE INVENTION

The present inventors have realized the need to provide an existingformat compliant 3D delivery system. Roughly described, the presentinvention provides a 3D delivery system that is backward compatible withexisting monoscopic delivery systems. This allows, for example, aBlu-ray video disc to be encoded under the Blu-ray format withStereoscopic data with the capability to be played on an ordinary BlueRay player and feed a 3D compatible display device for viewing by theconsumer. The present invention may be practiced by all other monoscopicformats including, for example, DVD formats, HD-DVD, the MPEG family,JPEG, etc.

In one embodiment, the present invention provides a method comprisingthe step of embedding a stereoscopic signal in a monoscopic-compatiblevideo infrastructure. In one embodiment, the embedded stereoscopicsignal comprises a video format where pixels from a first and a secondimage are stored in a “checkerboard” pattern within the monoscopiccompatible video infrastructure.

In one embodiment, the method further comprises the step of reducingpixels of a pair of stereoscopic images to be embedded in the videoinfrastructure. The step of reducing pixels comprises, for example,subsampling the pair of stereoscopic images and placing the subsamplesin a frame of the video infrastructure. The step of reducing pixelscomprises filtering the pair of stereoscopic images and mixing thefiltered images into a single frame of the video infrastructure. Thesubsampling may be performed, for example on coincident samples of theimages, related samples of the images, and/or the subsampling of a firstof the stereoscopic images is offset from the subsampling of a second ofthe stereoscopic images. In one embodiment, the locations of subsamplingof a first of the stereoscopic images is alternated with locations ofsubsampling of a second of the stereoscopic images.

The method further comprises, for example, the step of arranging thesubsamples in a lattice structure in a frame of the videoinfrastructure. Each subsample is placed, for example, in acorresponding location in the lattice structure occupied by thesubsample in the image from which it was subsampled. The monoscopiccompatible infrastructure comprises, for example, any of a Blu-raycompatible video format, and HD-DVD compatible video format, an Internettransmission format, a Direct-TV compatible video format, any of theMPEG family of formats, and an ATSC compatible video format.

In another embodiment, the present invention comprises a method,comprising the steps of, reducing pixels of a pair of stereoscopicimages, formatting the reduced pixels into an image pattern, andencoding the image pattern as a frame in a monoscopic video format. Thestep of reducing pixels comprises, for example, subsampling thestereoscopic images. The subsampling comprises, for example, samplinghalf of the pixels in a first of the stereoscopic images and subsamplinghalf of the pixels in a second of the stereoscopic images.

In various embodiments, the image pattern comprises a lattice pattern ofpixels comprising a checkerboard wherein “black” pixels of thecheckerboard comprise pixels derived from a first image of thestereoscopic pair of images and “white” pixels of the checkerboardcomprise pixels derived from a second image of the stereoscopic pair ofimages. A location of the pixels in the lattice pattern comprises, forexample, a coincident location in the image from which they werederived.

The method further comprises, for example, the step of decoding the“black” pixels into a first channel of a stereoscopic image and decodingthe “white” pixels into a second channel of the stereoscopic image.

The present invention may also be embodied in an encoding device,comprising, an input port configured to receive a bit pattern comprisingstereoscopic image pairs to be encoded, and an encoder configured toplace at least portions of each stereoscopic image pair into amonoscopic-compatible video encoded bit stream. The encoder isconfigured, for example, to produce a lattice structure comprisingportions of each image of a stereoscopic image pair. The latticestructure is produced, for example, by reduction and reordering ofpixels comprising alternately subsampling each of the images and placingthe subsamples in locations of a lattice structure corresponding to thelocations of the image from which they were subsampled. Themonoscopic-compatible video encoded bit stream comprises at least one ofan ATSC format, a Blu-ray format, an HD-DVD format, an existing videoformat, one of the MPEG family of formats, and another video format.

The invention may also be embodied as a decoding device, comprising, aninput port configured to receive a monoscopic formatted image signal, aprocessor configured to decode the monoscopic formatted image signal,and an image separator configured to extract a first image from eachframe of the decoded monoscopic formatted image signal and extract asecond image from each frame of the decoded monoscopic formatted image.The decoding device further comprises, for example, an image extractorconfigured to extract and expand each image to a full frame of a targetdevice. The target device comprises, for example, at least one of adisplay, an HDTV display, a cinema display, a cell phone display, acomputer display. The decoding device may further be part of a largermedia system comprising a display and the image expander is configuredto extract pixels for expansion in for each image from a checkerboardpattern within the decoded monoscopic formatted image.

The invention may also be embodied in any device or method that receivesa monoscopic formatted video and extracts and displays the multipleimages in any format. In one embodiment, the multiple images areextracted from a checkerboard pattern within the monoscopic formattedimage and then display as a 3D video.

Portions of both the devices and methods, and/or other embodiments, maybe conveniently implemented in programming on a general purposecomputer, or networked computers, and the results may be displayed on anoutput device connected to any of the general purpose, networkedcomputers, or transmitted to a remote device for output or display. Inaddition, any components of the present invention represented in acomputer program, data sequences, and/or control signals may be embodiedas an electronic signal broadcast (or transmitted) at any frequency inany medium including, but not limited to, wireless broadcasts, andtransmissions over copper wire(s), fiber optic cable(s), and co-axcable(s), etc.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the invention and many of the attendantadvantages thereof will be readily obtained as the same becomes betterunderstood by reference to the following detailed description whenconsidered in connection with the accompanying drawings, wherein:

FIG. 1A is a block diagram illustrating a basic structure according toan embodiment of the present invention;

FIG. 1B is a diagram of an encoder according to an embodiment of thepresent invention;

FIG. 1C is a diagram of a system for encoding and decoding imagesaccording to an embodiment of the present invention;

FIG. 1D is a diagram of system for encoding, decoding and displayingimages according to an embodiment of the present invention;

FIG. 2 is a diagram illustrating a system topology according to anembodiment of the present invention; and

FIG. 3 is an illustration of an image lattice structure according to anembodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Many current consumer stereoscopic systems require additionalprocesses—either as hardware devices or software applications—that theconsumer must purchase and install. This presents a significant barrierto market for the consumer and to the studios.

The present inventors have realized an implementation that allows thetransmission of a stereoscopic signal in a manner reusing the existingtelevision and video infrastructure by embedding the stereoscopic signalin a monoscopic-compatible signal. Using such an implementation, contentdistributors would then be able to deploy stereoscopic theatricalcontent into the home in order to exploit the marketing effort as closeto the theatrical release date as possible.

The present invention creates a method of distribution that can fit inthe current HD-DVD or Blu-ray, broadcast, and other distributioninfrastructures and also allows flexibility for future systems.Specifically, a future system could allow full bandwidth stereoscopicsignals to be transmitted while still maintaining backward compatibilitywith legacy stereoscopic devices. In this invention, a method ofencoding stereoscopic signals is combined with a number of image codingand picture structuring techniques in a novel way.

Referring now to the drawings, wherein like reference numerals designateidentical or corresponding parts, and more particularly to FIG. 1Athereof, there is illustrated a basic structure according to anembodiment of the present invention. The left and right images 101/102comprise 2 still images or two image streams representing the left andright images of a stereoscopic production. As illustrated at 103, theimages are encoded and packaged in a compliant means onto, for example,a standard HD-DVD or Blu-ray disc. The resulting stereoscopic dataappears as a standard image and can be presented to a display in itsencoded form. The display decodes the stereoscopic data back into itsleft and right image forms, where they are displayed (e.g., HDTV 104)where they appear as left and right images 105/106.

FIG. 1B is a diagram of an encoder according to an embodiment of thepresent invention. Stereoscopic images 110 are fed to an encoder 115.The images are packaged as a monoscopic format 120. The monoscopicformat comprises an existing monoscopic format, for example, a Blu-rayformat, an ATSC broadcast, or other formats.

FIG. 1C is a diagram of a system for encoding and decoding imagesaccording to an embodiment of the present invention. Stereoscopic images110 are fed to an encoder 115. The images are packaged as a monoscopicformat 120. The stereoscopic images are, for example, a series of leftand right channel images of a 3D video. The monoscopic format comprisesa monoscopic video signal where in each frame of the video is packagedwith the left and right channel images from each frame of the 3D video(monoscopic packaged 3D video signal).

A monoscopic decoder 125 includes at least one port for receiving amonoscopic video, and is fed the monoscopic packaged 3D video signal.The at least one port comprises, for example, any of an HDMI port,antenna port, S-Video connector, a cable connector, video/audiocomponent or composite connectors, network connector, 802.11 wireless,etc. In some embodiments related to transmission or carrier (e.g.,802.11), the monoscopic video signal is further packaged within therelated transmission or carrier transport protocol and an additionaldevice for extracting the monoscopic signal from the transport protocoland/or other protocols (not shown) is utilized. Ultimately, themonoscopic packaged 3D signal is received by the monoscopic decoder 125.

The monoscopic decoder 125 decodes the monoscopic packaged 3D signalaccording to the standard of the monoscopic format (e.g., Blu-ray, ATSC,etc). The decoded signal is a monoscopic video in which the images ofthe video are pattern decompressed images 130. That is, each image orframe of the decoded signal is a pattern image where the patterncomprises left and right channel images of the original stereoscopicimages 110.

A separator 135 follows a set or variable pattern in which the imagesare embedded in the decoded signal (pattern decompressed images 130) andextracts the original (original compressed and then decompressed) images(e.g., left and right channel images). The separator 135 may be aseparate device or may be packaged as part of a decoding system 140.

FIG. 1D is a diagram of system for encoding, decoding and displayingimages according to an embodiment of the present invention. Themonoscopic decoder 125 decodes the monoscopic packaged 3D signalaccording to the standard of the monoscopic format (e.g., Blu-ray, ATSC,etc). The decoded signal is a monoscopic video in which the images ofthe video are pattern decompressed images 130. That is, each image orframe of the decoded signal is a pattern image where the patterncomprises left and right channel images of the original stereoscopicimages 110.

A pattern recognizer 150 identifies an input signal as being either astandard monoscopic video encoded signal or a multi-image (e.g., stereo)encoded signal. The recognition may be performed, for example, bycomparing adjacent or otherwise related pixels in the decompressedpattern or constructing entire images based on a 3D encoded monoscopicpattern and comparing the resulting images.

Based on the recognition, the signal is then processed accordingly(e.g., the signal is routed to 2D display processing 155 for standardmonoscopic video or the signal is routed to 3D display processing 160for monoscopic packaged 3D), the results of which are then provided toand displayed on display 165. In one embodiment, the pattern recognizerand associated processing/processing devices are packaged together in adisplay (e.g., HDTV), as, for example, system 170. In anotherembodiment, the monoscopic decoder is also packaged together with thedisplay (system 175).

FIG. 2 is a diagram illustrating a system topology 200 according to anembodiment of the present invention. In FIG. 2, two images are presentedto the apparatus, one taken from the left perspective and the other fromthe right perspective both together forming a stereoscopic pair. Thesetwo images are, for example, low pass filtered 210A/210B and sub-sampled215A/215B by, for example, a factor of two. The sub-sampling operationproduces left and right samples (e.g., pixels) from the left and rightimages. The sub-sampling operation may be done in and may occur oncoincident locations on both left and right images or the sampledlocations can be offset from one image to the other. The preferredembodiment is to use offset sampling as shown in the lattice (orcheckerboard) structure described below.

The sampled image data is then arranged into a 3D image latticestructure by lattice structure device/processor 220. The 3D imagelattice structure is shown in FIG. 3. The left and right samples arearranged in alternating order within each line and then in the oppositeorder in the next line. This pattern is repeated throughout the lattice.The lattice structure aids in providing increased resolution in both thehorizontal and vertical dimensions upon image decoding with thedescribed system. Rather than reducing resolution in one dimension, theresolution is reduced in both dimensions but by a lesser amount. Theresult is a resolution reduction of approximately 0.7 fs rather than 0.5fs.

In one embodiment, the lattice structure changes, or alternates at apredetermined rate (e.g., once per frame). As shown in FIG. 3, the rowsof the lattice structure alternate between a L/R and a R/L patterns (aframe pattern of L/R-R/L). The entire structure may also be alternatedbetween different frame patterns. For example, a first frame comprisesthe L/R-R/L pattern and the second frame comprises a R/L-L/R pattern.

Once the lattice is populated with pixels in the prescribed manner, theimages are presented to an image encoder. The left and right imagesequences are temporally related and can make use of encoders such asMPEG encoders, JPEG encoders, or any other encoders used in videocompression. In one embodiment, colorspace conversion and chrominancesub-sampling are utilized during the encoding step (e.g., encoder 115).

Data produced by the image encoder 230 (e.g., a 3D encoded monoscopicpattern) is then packaged using a common transport mechanism bypackaging system 240. The transport mechanism is, for example, an MPEG-2transport stream or program stream. The net result of the packaging stepis to create a method where backward compatibility is preserved withdeployed systems. While the preferred embodiment is designed for HD-DVDor Blu-ray discs, the properly packaged data can be delivered using anydigital streaming method such as the Internet or conventional digitaltelevision broadcasting. Broadcasting can take the form of terrestrialbroadcasting, closed cable delivery systems, or satellite deliverysystems.

The decode side of the system comprises a reverse of the encode side.The delivered bit stream is presented to the decode apparatus by thedistribution system. The encoded image data is extracted using ademultiplexer and delivered to an image decoder. The image decoderconverts the compressed bitstream into the stereoscopic image data stillin the lattice structure. The lattice structure is then transmitted tothe display for final decoding into the left and right image pairs.

The decoding of the image is realized by removing the image data fromthe lattice structure and up sampling to the original image sizes (e.g.,3D processing 160 includes, for example, separates portions of thelattice structure related to each image and up-converts those samples tocreate the full images). The left and right image pairs are thenpresented to the viewer by, for example, displaying each image inseparate “flashes” on a display screen, interlacing the individualimages into a frame to be displayed, or other techniques.

The use of the lattice structure provides the stereoscopic functionalityto existing HD-DVD and Blu-ray players while not obsolescing theinstalled players. The system can employ messaging to configure theplayers to automatically provide the stereoscopic data to the display.These messages can be embedded in the bitstream in any number of waysincluding special SEI messages, MPEG private data, or as Java code inthe stream.

Although the present invention has been described herein with referenceto stereographic displays, the discussion herein also applies to thecoding, transmission, and decoding of multiple images in general. Infact, the present invention specifically includes embodiments withmultiple images. In one embodiment, a Blu-ray disk (or other medium)according to the present invention may include, for example, both 2D and3D versions of a movie or other production. In another embodiment, twoseparate 2D versions are included on the medium (e.g., left and rightviews) making the system compatible with systems with multiple decodersand do not wish to use the selected lattice or checkerboard pattern (forwhatever reason).

The bitrate between separate versions may allocated, for example,according to the complexity of each version. Complexity could beestimated given a variety of methods including MCTF preanalysis, basicencoding (i.e. intra) using the same quantization parameters and bitrateratio computation, or could just be based on user input or otherfactors. In a different embodiment, this could be done in a way toachieve a certain “average” distortion in either stream (this can be thesame or could be adjusted given a model or user input).

In describing preferred embodiments of the present invention illustratedin the drawings, specific terminology is employed for the sake ofclarity. However, the present invention is not intended to be limited tothe specific terminology so selected, and it is to be understood thateach specific element includes all technical equivalents which operatein a similar manner. For example, when describing a Blu-ray player, anyother equivalent device, such as DVD players, HD-DVD players, devicesthat deliver content (including memory devices, memory sticks, cameras,I-pods, etc), or other device having an equivalent function orcapability, whether or not listed herein, may be substituted therewith.

Furthermore, the inventors recognize that newly developed technologiesnot now known may also be substituted for the described parts and stillnot depart from the scope of the present invention. In fact the presentinvention specifically envisions application to new video standards andthe like not yet known or published. All other described items and otherequivalents, including, but not limited to sampling, filters,transmission protocols, storage protocols/formats, encoders, anddisplays (e.g., LCD, LCoS, Plasma, cinema projection, cinema projectionprocessors/servers, cinema storage devices, DLP devices, etc) shouldalso be considered in light of any and all available equivalents.

Portions of the present invention may be conveniently implemented usinga conventional general purpose or a specialized digital computer ormicroprocessor programmed according to the teachings of the presentdisclosure, as will be apparent to those skilled in the computer art.

Appropriate software coding can readily be prepared by skilledprogrammers based on the teachings of the present disclosure, as will beapparent to those skilled in the software art. The invention may also beimplemented by the preparation of application specific integratedcircuits or by interconnecting an appropriate network of conventionalcomponent circuits, as will be readily apparent to those skilled in theart based on the present disclosure.

The present invention includes a computer program product which is astorage medium (media) having instructions stored thereon/in which canbe used to control, or cause, a computer to perform any of the processesof the present invention. The storage medium can include, but is notlimited to, any type of disk including floppy disks, mini disks (MD's),optical discs, DVD, HD-DVD, Blu-ray, CD-ROMS, CD or DVD RW+/−,micro-drive, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs,DRAMs, VRAMs, flash memory devices (including flash cards, memorysticks), magnetic or optical cards, SIM cards, MEMS, nanosystems(including molecular memory ICs), RAID devices, remote datastorage/archive/warehousing, or any type of media or device suitable forstoring instructions and/or data.

Stored on any one of the computer readable medium (media), the presentinvention includes software for controlling both the hardware of thegeneral purpose/specialized computer or microprocessor, and for enablingthe computer or microprocessor to interact with a human user or othermechanism utilizing the results of the present invention. Such softwaremay include, but is not limited to, device drivers, operating systems,and user applications. Ultimately, such computer readable media furtherincludes software for performing the present invention, as describedabove.

Included in the programming (software) of the general/specializedcomputer or microprocessor are software modules for implementing theteachings of the present invention, including, but not limited to,sampling images, filtering image data, encoding multi-image data intopredetermined patterns, encoding multi-image data into a lattice (orcheckerboard structure), encoding a multi-image structure into amonoscopic image format, decoding monoscopic encoded data and expandingthe decoded data into multiple (e.g., stereo 3D) images and the display,storage, or communication of results according to the processes of thepresent invention.

The present invention may suitably comprise, consist of, or consistessentially of, any of element (the various parts or features of theinvention) and their equivalents as described herein. Further, thepresent invention illustratively disclosed herein may be practiced inthe absence of any element, whether or not specifically disclosedherein.

Obviously, numerous modifications and variations of the presentinvention are possible in light of the above teachings. It is thereforeto be understood that within the scope of claims to be included in asubsequently filed utility patent application, the invention may bepracticed otherwise than as specifically described herein.

What is claimed is:
 1. A non-transitory computer-readable storage mediumhaving stored thereon computer-executable instructions for executingwith one or more processors to perform: receiving encoded image datacomprising a sequence of coded frames, wherein a coded frame in thesequence of coded frames comprises samples of a first image and samplesof a second image packed together in a monoscopic video format, whereinthe monoscopic video format is determined using SEI messaging in theencoded image data; decoding the coded frame to generate a decodedframe; unpacking the decoded frame based on the monoscopic video formatto generate decoded samples of the first image and decoded samples ofthe second image, wherein the decoded samples of the first image and thesecond image are arranged in accordance with the monoscopic video formatin a lattice checkerboard pattern wherein “black” pixels in the latticecheckerboard pattern comprise sampled pixels of the first image and“white” pixels in the lattice checkerboard pattern comprise sampledpixels of the second image, wherein sampling the “black” pixels of thefirst image comprises: on even rows, sampling only even columns of thefirst image, and on odd rows sampling only odd columns of the firstimage, and sampling the “white” pixels of the second image comprises: oneven rows, sampling only odd columns of the second image, and on oddrows sampling only even columns of the second image; upsampling thedecoded samples of the first image to generate a first output image; andupsampling the decoded samples of the second image to generate a secondoutput image.
 2. The non-transitory computer-readable storage medium ofclaim 1, wherein the first image and the second image represent twoviews of a stereoscopic image.
 3. The non-transitory computer-readablestorage medium of claim 1, wherein the coded frame is encoded accordingto at least one of a Blu-Ray compatible video format, a DVD compatibleformat, an MPEG format, and an ATSC compatible video format.